Unintended Consequences – The Hidden Cost of Innovation

Occasionally, a breakthrough inadvertently creates a bigger problem than it solves, leaving us to wonder: is the price of unintended consequences worth the innovation?

When the Internet pioneer, Lou Montulli, developed the browser cookie in 1994, his intention was to protect user privacy. Little did he know that decades later, this small snippet of code would allow corporations to track users around the web, and his brainchild would be among the most controversial inventions of the digital age. Arguably, Lou couldn’t have foreseen this ironic twist of fate, but the progress the cookie enabled undoubtedly came with an unexpected cost.

So how do you avoid these hidden pitfalls of innovation? Is it even possible? I caught up with Lou, a trailblazer featured in our Hidden Heroes series, for a candid chat on how the cookie came about, its exploitation, and whether that could have been prevented.

Igniting the sparks of innovation

Whatever your passion, it’s important to have the courage to look for opportunities and sparks of innovation. For Lou, the first flicker of a groundbreaking idea came while he was a student at the University of Kansas in the early nineties. The Internet was very much in its infancy and used numbers to navigate documents, but Lou could see the potential for hypertext as a more natural way of navigating.

“The spark of innovation was when I saw HyperRez – and I knew about Gopher – I thought, ‘well, what if I just put them together?’”

– he recalls. And so the idea for Lynx, one of the earliest web browsers, was born.

Lou wasn’t alone in his efforts, though. At the same time, an entire web development movement was forming around the world. “I quickly realized that we would be a stronger community if we worked together,” says Lou. “So I made Lynx compatible with the web and, at that point, it became a true worldwide web browser. Over time, other people started to work on a Windows and a Mac version along with other versions of web browsers, and it was then that the web started to really gain momentum.”

Lou has since continued to be a strong advocate of open standards and open-source software.

“Open standards allow everyone to play, which causes a strong and fast innovation cycle,”

– he says of the modern-day competition between browsers. “When everyone controls the technology and there are multiple competitors in the space, it's good economics. They will all advance at a much faster rate, and we all win when that happens.”

Creating the cookie

By the mid-nineties, Lou had become one of the founding engineers at Netscape Communications. There, he ignited his second major spark of innovation – the browser cookie.

In the early days of the web, there was no memory between requests. “When you go to a site using HTTP, your browser connects, grabs a document, and then disconnects,” explains Lou.

“Then when it wants the next document, it connects and disconnects, and there's nothing there that identifies the browser in any way.” This was fine for reading documents but meant that anything more complex, such as putting items in a shopping cart, wasn’t possible.

“The cookie was designed to do two things,” says Lou. “To add state to a particular website, so you could create shopping carts and other applications from it, and to preserve your anonymity across many other websites […] We didn't want to create a tracking mechanism that allowed you to be tracked across websites.”

Paving the way for tracking: How the cookie crumbled

Ironically, despite Lou’s intention for the cookie to preserve user privacy, within just a few years, it was being exploited by large corporations. Using a combination of technologies, marketers could track visitors across multiple sites, monitor their online behavior, and serve targeted ads.

“It's only through collaboration with other websites that it can be done,” explains Lou. “You take an advertising network like DoubleClick, which was the first to exploit this, and they worked with about a hundred different sites and convinced them to put a particular image on their website.”

These embedded images pointing back to DoubleClick enabled the creation of third-party requests and empowered advertisers to track the sites a user had visited. “It's that scenario that allows advertising to be ten or 20 times more effective and, in the end, profitable. It really allowed content creators to monetize their content in a way that made sense,” says Lou.

Therefore, Lou and his team found themselves in a dilemma when they realized a leak was occurring. “If we completely removed the third-party cookie, which was certainly technically possible, it would have severely diminished the ability for advertising to work on the web,” he says. “[Instead], we created a bunch of tools within the browser to control cookies, make sure you know they're there, and clear them.”

Thanks in no small part to its targeting capabilities, today, digital ad spending in the US alone amounts to more than $200 billion, and it’s expected to continue rising.

Innovating for a different time

Arguably, Lou could never have identified the tracking vulnerability before it became an issue. In his words,

“it’s not possible to be imaginative enough to come up with all of the possibilities.”

The web was a very different place back then, mainly providing a mechanism for browsing documents, and there was very little commerce happening. Given this scenario, it would have been tough to predict how the landscape would change in the future. Even so, Lou and his colleagues did try.

“After we came up with the initial concept, we spent a lot of time thinking about ways in which it could be made better or how it could be exploited, and this particular scenario didn't come to mind,” says Lou.

Avoiding the unintended costs of innovation

Over their thirty-year history, cookies have gained notoriety, but is it fair to hold innovators responsible for the impact of their work when things take an unexpected turn? Could software creators better anticipate the ways their ideas might be manipulated? In the featured Hidden Heroes story, Lou shared,

“The law of unintended consequences definitely applies here. I do believe people should think long and hard about the consequences of what they are building.”

At the same time, though, it’s impossible to know how things would have turned out had Lou never written those significant lines of code. I would argue that with the interest in advancing the Internet in the nineties, we may have eventually reached the same outcome by a different means.

Certainly, nowadays, the cookie is just one of many ways corporations can track us, including more invasive methods like fingerprinting and domain spoofing. “If the solution is simply to disable cookies, the advertising networks will move to something else that you will not have control over,” says Lou. “I would advocate that the devil you know is better than the devil you don’t.”